Resolve OpenSSL 4.0 build issues by maskit · Pull Request #13476 · apache/trafficserver

maskit · 2026-08-03T16:59:38Z

Summary

Continues #13440 ("Resolve OpenSSL 4.0 build issues"), fixing the remaining CI failures on Ubuntu 20.04 and FreeBSD 13.1 (OpenSSL 1.1.1) flagged in review.

X509_NAME_get_index_by_NID() takes a non-const X509_NAME * on OpenSSL 1.1.1 but a const X509_NAME * on OpenSSL 3.0+ and post-constify OpenSSL/BoringSSL. The original PR widened several X509_NAME-typed variables to const to build against the newer, constified signature, which broke the older signature at three call sites:

src/tscore/X509HostnameValidator.cc — validate_hostname()
src/iocore/net/SSLUtils.cc — SSLMultiCertConfigLoader::load_certs_and_cross_reference_names()
plugins/experimental/txn_box/plugin/src/ts_util.cc — ssl_value_for()

Each call site now does const_cast<X509_NAME *>(...), matching the existing precedent in OCSPStapling.cc for the same kind of OpenSSL-version const mismatch.

Test plan

Built tscore, inknet, and txn_box targets locally against BoringSSL — all compile clean
Ran cmake --build build -t format — no additional changes
CI to confirm the Ubuntu 20.04 / FreeBSD 13.1 (OpenSSL 1.1.1) jobs that failed on Resolve OpenSSL 4.0 build issues #13440 now pass

JosiahWI · 2026-08-03T17:15:03Z

Did you consider the decltype workaround discussed in #13440? I would like to see an explicit justification for opting to use the less type safe const_cast approach.

maskit · 2026-08-03T17:59:14Z

[approve ci]

maskit · 2026-08-03T18:02:20Z

@JosiahWI Thanks for the reminder. The use of auto made it a little cleaner. It didn't completely remove the const_casts, but I think this is ok for 10.2. After landing this, we should do cleanup that removes 1.1.1 support on master.

JosiahWI · 2026-08-03T19:13:54Z

I think decltype will preserve the CV qualification on the type, unlike auto.

JosiahWI · 2026-08-03T19:17:06Z

Oh, I see. The auto keyword also determines the correct constness in most cases. This looks good.

Copilot

Pull request overview

This PR continues the effort to keep ATS building across OpenSSL versions (OpenSSL 1.1.1 through 4.0+/constified APIs and BoringSSL) by tightening const-correctness around X509/ASN1 access and updating code to use supported accessor functions instead of direct structure field access.

Changes:

Replace direct ASN1_STRING/ASN1_BIT_STRING field access (->data, ->length, ->type) with accessor APIs like ASN1_STRING_get0_data(), ASN1_STRING_length(), and ASN1_STRING_type().
Update several helper signatures and local variables to const where the OpenSSL APIs now expose const data (e.g., X509_NAME, X509_NAME_ENTRY, ASN1_STRING).
Adjust certificate subject-name manipulation in certifier to operate on a duplicated X509_NAME and set it back onto the certificate.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/tscore/X509HostnameValidator.cc	Uses ASN1 accessor APIs for hostname checks and adjusts CN parsing types.
src/iocore/net/SSLUtils.cc	Makes `asn1_strdup()` accept `const ASN1_STRING*` and propagates const correctness through subject CN extraction.
src/iocore/net/SSLNetVConnection.cc	Updates debug helper to accept `const X509_NAME*`.
src/iocore/net/OCSPStapling.cc	Replaces `ASN1_BIT_STRING` direct field access with ASN1 accessor APIs for digesting and const cleanup.
src/api/InkAPI.cc	Uses const-correct `X509_NAME_ENTRY` / `ASN1_STRING` pointers when extracting CN during cert update.
plugins/experimental/txn_box/plugin/src/ts_util.cc	Makes `ssl_value_for()` accept `const X509_NAME*` and uses a `const_cast` for OpenSSL 1.1.1 compatibility at one call site.
plugins/experimental/sslheaders/expand.cc	Makes subject/issuer names const and replaces signature byte access with ASN1 accessor APIs.
plugins/experimental/cert_reporting_tool/cert_reporting_tool.cc	Treats subject name as `const X509_NAME*`.
plugins/certifier/certifier.cc	Duplicates and sets the subject name before adding CN to avoid mutating internal OpenSSL-owned structures directly.

cmcfarlen · 2026-08-03T21:39:48Z

I had independently worked up a fix for this (#13482) before seeing this PR — standing that down in favor of yours. Handing over what I found in case it's useful, since some of it isn't covered here yet.

The diagnosis and the three X509_NAME_get_index_by_NID call sites match what I landed on exactly. The one thing worth flagging: neither CI nor a BoringSSL build can validate the OpenSSL 4 side of this. The Fedora CI job is Fedora 43 (OpenSSL 3.5), and per your own table in #13440 BoringSSL returns non-const from X509_get_subject_name — i.e. the same shape as 1.1.1. So a green CI run plus a BoringSSL build confirms the 1.1.1 regression is fixed, but can't exercise any of the const-return changes that OpenSSL 4 introduces.

I built this branch in a Fedora 45 / OpenSSL 4.0.1 container (BUILD_EXPERIMENTAL_PLUGINS=ON, ENABLE_CRIPTS=ON, BUILD_REGRESSION_TESTING=ON, ninja -k 0 to get past the first failure). Remaining errors:

File	Errors
`include/cripts/Certs.hpp`	124
`src/cripts/Certs.cc`	4
`plugins/lua/ts_lua_client_request.cc`	4
`plugins/lua/ts_lua_client_cert_helpers.h`	3
`src/iocore/net/unit_tests/test_SSLDHParams.cc`	1

Specifically:

plugins/lua — get_x509_name_string(X509_NAME *) needs a const X509_NAME * param (it only calls X509_NAME_print_ex, which is const on every version), and ts_lua_client_cert_helpers.h reads sig->length / sig->data directly, which breaks now that ASN1_STRING is opaque. Note this one builds by default — plugins/lua is gated on if(USE_LUAJIT), and Fedora 45 has luajit — so it's not an opt-in path.
src/cripts — Certs.cc has the same opaque-ASN1_STRING issue in Signature::_load and _write_ip_address. ENABLE_CRIPTS defaults OFF so this is lower priority.
test_SSLDHParams.cc — make_cert_and_key mutates the subject name in place. Once the accessor returns const you have to dup, then call both X509_set_subject_name and X509_set_issuer_name, or the generated self-signed cert silently loses its CN.
Also broken on OpenSSL 4 but outside a default build: example/plugins/c-api/client_context_dump/client_context_dump.cc and tests/tools/plugins/ssl_client_verify_test.cc.

Two notes on approach, take or leave:

const_cast handles the three call sites fine, but it can't handle Cripts. CertBase::X509Value::_load_name(X509_NAME *(*getter)(const X509 *)) takes a function pointer, and X509_get_subject_name's whole type changes in 4.0 — casting a function pointer and calling through it is UB. X509_getm_notBefore/notAfter also moved const the opposite direction (const X509 * in 1.1.1, X509 * in 4.0), so no single hardcoded spelling works there either. What worked for me was deducing the type from the accessor — auto * for locals, decltype(&X509_get_subject_name) for parameter and function-pointer types — which needs no casts and no version macros.
The certifier.cc change (carried from Resolve OpenSSL 4.0 build issues #13440) leaks the duplicated name on the X509_NAME_add_entry_by_txt failure path — only the X509_set_subject_name path frees it — and doesn't null-check X509_NAME_dup.

Diffs for all of the above are on cmcfarlen:ossl4-compat (#13482) if you want to cherry-pick or copy from them; verified green on Fedora 45/OpenSSL 4.0.1, Fedora 43/OpenSSL 3.5.7, and Ubuntu 20.04/OpenSSL 1.1.1f + clang-12, with test_tscore and test_net passing on all three. Happy to just close mine and let you carry it however you prefer.

…t-iness

OpenSSL 1.1.1 declares the first argument as non-const while newer releases declare it const, so building against different versions failed depending on which signature was in effect. Add a const_cast at each call site to keep the const-qualified variables introduced by this PR buildable against both, matching the existing precedent in OCSPStapling.cc.

X509_get_subject_name and X509_NAME_get_index_by_NID have their argument constness changed together across OpenSSL versions, so a local variable declared with auto tracks whatever type is correct for the OpenSSL version in use, without a cast. This applies only to the two purely local variables; ts_util.cc ssl_value_for keeps its const_cast since its parameter type is shared across multiple callers.

X509HostnameValidator.cc cast ASN1_STRING_get0_data return value to non-const before an ats_strndup call that only wants const char *; drop the const instead of adding it back needlessly. certifier.cc leaked the duplicated X509_NAME on the X509_NAME_add_entry_by_txt failure path, and never checked X509_NAME_dup for allocation failure.

CertBase::X509Value took hardcoded function pointer types for X509_get_subject_name, X509_get_issuer_name, X509_getm_notBefore, and X509_getm_notAfter, but those accessors change constness in different directions across OpenSSL versions, so no single hardcoded signature builds everywhere. Deduce the parameter type from the actual accessor via decltype instead. Signature::_load and _write_ip_address also read ASN1_STRING fields directly, which breaks once the struct is opaque; switch to the accessor functions used elsewhere in this codebase.

get_x509_name_string only reads through the name via X509_NAME_print_ex, so accept a const X509_NAME * to match callers that pass X509_get_subject_name/X509_get_issuer_name results directly. get_x509_signature_string read the ASN1_STRING struct fields directly, which breaks once the struct is opaque; use ASN1_STRING_get0_data/ASN1_STRING_length instead.

make_cert_and_key mutated the X509_NAME returned by X509_get_subject_name in place, which stops compiling once that accessor can return const, and was already fragile since the returned name is only a view into the certificate internal state. Duplicate it, add the CN to the duplicate, and set it back as both subject and issuer name since this is a self-signed certificate.

client_context_dump.cc, verify_cert.cc, and ssl_client_verify_test.cc all held X509_get_subject_name/X509_NAME_get_entry/ X509_NAME_ENTRY_get_data results in hardcoded non-const locals or parameters, which stops compiling once those accessors return const. Switch to auto for the local variables and const for the debug_certificate parameter, matching the read-only usage in each case.

ssl_value_for is shared by four callers, each already deducing its X509_NAME pointer type with auto from X509_get_subject_name or X509_get_issuer_name, so unlike the other three call sites fixed earlier in this series, a single hardcoded parameter type cannot track the underlying accessor across OpenSSL versions. Templating the parameter on the callers deduced type removes the cast entirely.

Bare decltype(&X509_get_subject_name) in a parameter list reads poorly at each of the four call sites; name each getter type once via using so the declarations and out-of-line definitions just say what kind of accessor they take.

certifier.cc freed the duplicated X509_NAME manually on every path; switch to a scoped_X509_NAME unique_ptr matching the file existing scoped_X509/scoped_EVP_PKEY/scoped_SSL_CTX aliases so no path can forget to free it. ts_util.cc templated ssl_value_for just to defer the parameter type to the caller; decltype(X509_get_subject_name(nullptr)) deduces the same pointer type directly without turning it into a template. Also drop the last C-style cast this series introduced in X509HostnameValidator.cc in favor of reinterpret_cast.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Suppressed comments (2)

plugins/certifier/certifier.cc:411

X509_NAME_add_entry_by_txt() expects a const unsigned char *, but this code uses a non-const cast (unsigned char *)commonName.c_str(). Avoid casting away constness here (and avoid C-style casts) to keep the code const-correct across OpenSSL variants.

  if (X509_NAME_add_entry_by_txt(n.get(), "CN", MBSTRING_ASC, (unsigned char *)commonName.c_str(), -1, -1, 0) != 1) {

plugins/experimental/txn_box/plugin/src/ts_util.cc:1119

The PR description says the OpenSSL 1.1.1 vs 3.x+ X509_NAME_get_index_by_NID() const-mismatch is handled by adding const_cast<X509_NAME*>(...) at the affected call sites, but this change instead introduces a decltype(X509_get_subject_name(nullptr)) alias and does not add any such const_cast (and none exist in-tree). Please update the PR description to match the implemented approach, or adjust the code to match the documented fix.

  using X509_NAME_ptr = decltype(X509_get_subject_name(nullptr));

  TextView
  ssl_value_for(X509_NAME_ptr name, int nid)
  {
    if (int loc = X509_NAME_get_index_by_NID(name, nid, -1); loc >= 0) {

maskit · 2026-08-03T23:33:36Z

+    unsigned char     *astr;
    str = X509_NAME_ENTRY_get_data(X509_NAME_get_entry(name, i));
    // Convert to UTF-8
    astrlen = ASN1_STRING_to_UTF8(&astr, str);


Confirmed real, but pre-existing on master since the function's original addition in 2015, unrelated to this PR's OpenSSL 4 changes. Filed as #13483.

maskit · 2026-08-03T23:33:38Z

+    const int              pos              = X509_NAME_get_index_by_NID(X509_get_subject_name(cert.get()), NID_commonName, -1);
+    const X509_NAME_ENTRY *common_name      = X509_NAME_get_entry(X509_get_subject_name(cert.get()), pos);
+    const ASN1_STRING     *common_name_asn1 = X509_NAME_ENTRY_get_data(common_name);
    char *common_name_str = reinterpret_cast<char *>(const_cast<unsigned char *>(ASN1_STRING_get0_data(common_name_asn1)));
    if (ASN1_STRING_length(common_name_asn1) != static_cast<int>(strlen(common_name_str))) {


Confirmed real, but pre-existing on master since 2024, unrelated to this PR's OpenSSL 4 changes. Filed as #13484.

maskit · 2026-08-04T00:22:37Z

[approve ci autest 2]

maskit · 2026-08-04T02:37:12Z

I've updated the change and addressed all the comments. New code does not use const_cast at all now. I did see const_casts around the changes but those are just few of 500+. Those should be fixed separately.

maskit · 2026-08-04T02:45:20Z

@jeredfloyd I added changes on top of your commits to cover the entire source tree. I'd appreciate it if you could test this change for Fedora 45.

JosiahWI

This looks great to me. Will wait for @jeredfloyd to check before approving. I have one question about the certifier logic.

jeredfloyd

Looks good to me, and confirmed this builds on Fedora 45 and EPEL 10. (I'm old and prefer explicit types over inferred wherever possible, and this looks like a reasonable balance.)

JosiahWI

Approved!

maskit added the TLS label Aug 3, 2026

maskit added this to the 11.0.0 milestone Aug 3, 2026

maskit added this to ATS v10.2.x Aug 3, 2026

maskit marked this pull request as ready for review August 3, 2026 19:46

Copilot AI review requested due to automatic review settings August 3, 2026 19:46

Copilot started reviewing on behalf of maskit August 3, 2026 19:59 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

Comment thread src/tscore/X509HostnameValidator.cc

Comment thread plugins/certifier/certifier.cc

cmcfarlen mentioned this pull request Aug 3, 2026

Resolve OpenSSL 4.0 build issues #13482

Closed

jeredfloyd and others added 5 commits August 3, 2026 16:21

Resolve OpenSSL 4.0 build issues - accessors for ASN1_STRING and cons…

c683d55

…t-iness

Ran clang-format to fix formatting

d8a37f9

cmcfarlen requested a review from JosiahWI August 3, 2026 22:27

maskit added 7 commits August 3, 2026 16:44

Alias the Cripts X509 getter decltypes for readability

45809f6

Bare decltype(&X509_get_subject_name) in a parameter list reads poorly at each of the four call sites; name each getter type once via using so the declarations and out-of-line definitions just say what kind of accessor they take.

Copilot AI review requested due to automatic review settings August 3, 2026 23:14

maskit force-pushed the pr-13440-openssl4-build-fix branch from d31a219 to 01b9317 Compare August 3, 2026 23:14

Copilot started reviewing on behalf of maskit August 3, 2026 23:15 View session

Copilot AI reviewed Aug 3, 2026

View reviewed changes

This was referenced Aug 3, 2026

validate_hostname() returns -1 on UTF-8 conversion failure, which converts to true in its bool return type #13483

Closed

TSSslServerCertUpdate() can crash or read out of bounds extracting a certificate CN #13484

Open

maskit self-assigned this Aug 3, 2026

maskit added the Build work related to build configuration or environment label Aug 3, 2026

JosiahWI reviewed Aug 4, 2026

View reviewed changes

Comment thread plugins/certifier/certifier.cc

Comment thread plugins/certifier/certifier.cc

Comment thread example/plugins/c-api/verify_cert/verify_cert.cc

jeredfloyd approved these changes Aug 5, 2026

View reviewed changes

jeredfloyd mentioned this pull request Aug 5, 2026

Resolve OpenSSL 4.0 build issues #13440

Closed

JosiahWI approved these changes Aug 5, 2026

View reviewed changes

maskit merged commit bdefaf9 into apache:master Aug 5, 2026
15 checks passed

github-project-automation Bot moved this to For v10.2.0 in ATS v10.2.x Aug 5, 2026

Uh oh!

Conversation

maskit commented Aug 3, 2026

Summary

Test plan

Uh oh!

JosiahWI commented Aug 3, 2026

Uh oh!

maskit commented Aug 3, 2026

Uh oh!

maskit commented Aug 3, 2026

Uh oh!

JosiahWI commented Aug 3, 2026

Uh oh!

JosiahWI commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

cmcfarlen commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

maskit Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

maskit Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

maskit commented Aug 4, 2026

Uh oh!

maskit commented Aug 4, 2026

Uh oh!

maskit commented Aug 4, 2026

Uh oh!

JosiahWI left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeredfloyd left a comment

Choose a reason for hiding this comment

Uh oh!

JosiahWI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

JosiahWI left a comment •

edited

Loading